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THE STUDY OF SCHOOL 
EFFECTIVENESS : MICHIGAN 



SECTION I 
INTRODUCTION 

The purpose of this report is to provide a summarized description 
of the 1974-75 Michigan Cost-Effectiveness Study and its findings. This 
report is comprised of seven sections. In addition to this Introduction, 
there are: 

Section II - Purpose of 1974-75 Study 
Section III - Study Design for 1974-75 
Section IV - Description of 1974-75 Sample 
Section V - Cost Analyses 
Section VI - Effectiveness Analyses 

Section VII - Summary, Conclusions, and Recommendation 
As was the case with the first year of the study, 1973-74, the 
1974-75 study was restricted to compensatory education reading programs. 
The study was an effort to develop and implement evaluation techniques 
which can determine what educational practices bring about changes in 
student behavior and what costs are associated with those practices. 
Information about those educational practices and their associated costs 
would provide a rationale for planning. This rationale for planning 
would provide the base for educators to develop new programs and modify 
existing programs to improve services to students. 

The Cost-Effectiveness Study focused upon educational variables 
which could be changed or controlled by educational systems. Variables 
such as race, social economic status, level of parental education, and so 
forth, which cannot be readily controlled or modified by an educational 
system were not examined. It was the intent of the study to examine 
those aspects of educational policy and practices which could be changed 
to bring about a higher quality of education for students. 



The term program effectiveness, as used in this study, includes 
a consideration of both program success and activities associated with 
the program. Program success is attained when the objectives of a program 
are attained, i.e., an increase in student reading achievement. Program 
effectiveness required further investigation of the question, **Why was 
the program successful?'* To be termed effective, the activities of the 
program must contribute to the success of the program, i.e., there is 
a strong indication that the activities brought about the achievement 
of the objectives. 

Both successful and unsuccessful reading programs were included in 
the study. The question might be asked, "Why would someone want to look 
at unsuccessful programs if they were trying to find out what makes a 
successful program?" The answer to that question is fairly simple. If 
a person were to look at a group of successful reading programs, they 
would probably find that all of those successful programs have a program 
director, provide reading teachers, and provide .i assortment of reading 
materials. However, if that same person were to look at unsuccessful 
reading programs, they would probably find that those programs also have 
a program director, have reading teachers, and provide an assortment of 
reading materials. The information obtained from examining the 
unsuccessful reading programs would show that having a program director, 
the presence of reading teachers, and an assortment of reading materials 
are not unique aspects of successful reading programs. Discovering what 
is unique about successful reading programs can only be accomplished by 
looking at both successful and unsuccessful programs. 

7 
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SECTION II 
PURPOSE OF 1974-75 STUDY 



The purpose of the 1974-75 study efforts was the continued 
development of the analytical techniques reflected in the cost- 
effectiveness model. However, a critical part of this continued 
development was the cross-validation effort focusing upon the 
variables identified and reported in the executive summary^of the 
1973-74 study. Thus, Section 6 of this report, which presents the 
results of this year's study, reports first on the results of this 
cross-validation effort. This order of reporting reflects the 
importance of cross-validation since without some evidence upon which 
to anchor the overall findings, evidence that involves the demonstration 
of significant results over more than one year of the study, it might 
vjell be argued that any other findings are greatly lessened in their 
impact . 

Following, in importance, the cross-validation of the reported 
results of last year's effort is the identification of new variables 
which relate to achievement. Additionally, much of the discussion and 
analysis of Section V, dealing with the 1974-75 cost analysis, also is 
concerned with identifying new significant variables. 

The bulk of tlie work reported in Section V details the extended 
relationships investigated between cost and achievement. This extension 
is one both of a refinement nature where data management and reduction 
techniques were involved and of a broadened analysis nature where costs 
provided from various funding sources are analyzed this year, a series 
of analyses not possible using last year's (1973-74) data. 

-^Michigan Cost-Effectiveness Study: An Executive Summary, Michigan 
Department of Education, April, 1975. ^ 



The last of the three major purposes of the 1974-75 effort, the 
investigation of the direction of the relationships between achievement 
and various identified variables, is addressed in Section VI. Time 
constraints and the volume of data, with the concommitant data management 
needs, prevented all of the possible analyses from being completed. 
However, the development of the path models presented in Section VI do 
represent a major step in identifying the nature of ^he significant 
relationships between various variables and reading achievement. 

The 1974-75 study effort was designed to meet these purposes; the 
extent to which each purpose has been met is well reflected in the pages 
that follow. The model, first began in the spring of 1972 and further 
developed through the 1973 pilot and the 1973-74 study effort, has 
seen still further developmental progress through the 1974-75 study 
effort. Instruments have been modified to remove uninteresting variables 
and pursue interesting areas in greater depth and further. Data management 
and analysis techniques have been simplified and refined. The cost 
analysis methodology has been critically examined, expanded to include 
estimates of costs provided from various funding sources, and refined 
through an overall improvement of the consistency of data collection and 
management techniques . 
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SECTION III 
STUDY DESIGN FOR 1974-75 

For the 1974-75 study year, a number of changes in the overall 
study design were carried out. During the previous study year one 
building per study site had been included in the sample. This building 
w.is the outlier building for that site^ high achieving out3.ier for 
high achieving sites and low achieving outlier for low achieving 
sites. For the 1974-75 study year, two buildings per study site were 
included in the sample, both high and low achieving outliers from each 
site regardless of whether the site was selected as a high or low 
achieving site. 

The move from one to two buildings per site had a direct impact 
on at least one aspect of the study sample for the 1974-75 year, A 
number of the districts included in the 1973-74 sample were districts 
which had only one elementary building. For the 1974-75 study year, 
though, no districts having only one elementary school were included 
in the study due to the requirement of two buildings per site. Thus, 
the study sample for the 1974-75 study year tended to include districts 
which on the average were larger than the previous year's study sample. 

Selection of Sites and Scheduling Contacts 

Following the overall ditfiensions intended for the 1973-74 study, 
the site selection process beg<isri with identifying 25 districts in 
Michigan highly successful in their compensatory education reading 
programs and 25 that were highly unsuccessful. Thirty six of these 
50 districts were to be included for their Title I programs; fourteen 
for their Chapter 3 programs. 

10 
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Similar to what was done in the previous year's effort, selection 
of the successful and unsuccessful Title I sites began by reviewing 
the 1973-74 Title I evaluation reports of the approximately 500 LEAs 
(Local Education Agencies) and LEA co-ops to rank the 36 highest achieving 
and 36 lowest achieving programs based on the average months gained per 
month in the program in reading achievement. The following guidelines 
were followed: 

1) adequate program description had to be available for the 
district ; 

2) student population turnover for the district had to have been 
less than 40 percent; 

3) the district had to have at least two schools; 

4) each school had to have at least 2 grades between grades 2-6 
with 15 or more compensatory education students in each grade, 
thus requiring a minimum of 20 students in the district's program; 

5) reading test results on MDE acceptable reading tests were used 
for comprehension if available, for vocabulary if comprehension 
results were not available, or for total score if neither sub-test 
result was available; 

6) the pre-test must have been given prior to December 1973; 

7) at least 7 months had to have elapsed between pre and post tests; 

8) where no specific date was specified for the month in which 

the tests were reported to have been administered, it was assumed 
that the test was administered in the middle of the month; 

9) the program length was rounded to the nearest 0.5 month; 

10) where specific test dates were provided it was assumed that 
the days of the month numbered from 1-10 is the beginning of 
the month, 11-15 is the middle of the month, and 21-31 is 
the end of the month; 

11) months gained per month in the program was calculated for 
any grade level from grade 2 through 6 where at least 5 
students participated in testing; 

12) these grade averages, in turn, were averaged to form a program 
average; and 

13) month per month gains were recorded to the nearest hundredth. 

11 
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Use oi these guidelines allowed the 72 Title I programs, 36 at each end 

of the achievement spectrum, to be identified, from which the desired level 

of 18 programs at each end would be selected. 

Paralleling this detailed effort for the Title I programs, the 14 
highest and 14 lowest achieving Chapter 3 districts were also identified. 
As was necessary last year, percentage of accomplishment was used to 
determine these performance levels rather than month per month gains. 
Reading achievement results were used to rank all Chapter 3 districts in 
terms of the percent of their Chapter 3 students reaching at least the 75% 
level of accomplishment. This ranking was then used to identify the desired 
Chapter 3 sites. 

Both the Title I and Chapter 3 pools of sites were deliberately 
selected as b<^ing double the size needed to allow replacement of sites 
initially selected where additional program stability criteria caused 
a site to be dropped from the sample. The initial 50 sites, reflecting 
the dimension indicated in the opening paragraph of this discussion, were 
randomly selected from the pool of Title I and Chapter 3 sites just 
described. 

Additional school level screening criteria were then applied to 
these sites through written and telephone contact. These criteria 
i Included: 

1) the compensatory education program (Title I or Chapter 3) 
was in existence by the fall of 1973; 

2) the program had the same key persons (e.g., reading coordinator) 
as in 1973-74; or the same key persons provide the same services 
to the program as were provided the previous year, even though 
these persons may hold different titles or be in different locations 

3) the scliool building had the same principal as in 1973-74; 

4) teacher turnover in the building was less than 40 percent; 

12 
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5) there were at least five compensatory education students per 
participating grade level; and 

6) the materials used were essentially those used in the previous 
s,chool year. 

Through these written and telephone contacts, the selected districts 
were asked to identify their three highest achieving schools meeting the 
above listed criteria and their three lowest achieving schools meeting 
these >-ame criteria. Direct contact with the identified buildinpis hen 
allowed verification of the information provided by district level 
correspondents and, eventually, specification of the two schools to be 
studied per site. 

Problems of proji;ram stability in the low achieving sites, similar 
to those encountered last year, prevented the desired 50 sites from being 
identified and included in the current year's study. Table 1 shows the 
results of the selection process just described. /\ total of 96 schools 
from 48 sites were thus selected including 36 schools from 18 high 
achieving Title I sites. 36 Trom 18 low achieving Title I sites, 14 from 
7 high achieving Chapter 3 sines, and 10 from 5 low achieving Chapter 3 
s i tes . 

TABLE 1 

SITE/SCHOOL SELECTION FOR 1974-75 STUDY 

~ TITLE I SITES CHAPTER 3 SITES TOTAL 

No, of No. of No. of No. of No. of No. of 
Sites Schools Sites Schools Sites Schools 

High Achieving Sites 18 36 7 14 25 50 

Low Achieving Sites 36 _5 i2 23 46 

TOTAL 36 72 12 24 48 96 



S election and Training of Data Collectors 

The data collection team for the 1974-75 study included six members. 

The on-site data collection activities of these individuals was supervised 

by the project's data coordinator. The individual serving as data 

coordinator was the same person who held this position during the previous^ 

year of the study; and four of the six data collectOL^s were also data 

collectors for the study last year. 

A one day training session was provided for all members of the data 

2 

team. This session was supervised by Education Turnkey Systems, Inc. 
principal investigator and conducted jointly by the project director and 
the data coordinator. 

This training session covered a number of topics including: 

1) a summary of the progress of the 1974-75 study since December 
of 1974; 

2) a discussion of the history of the project through the 1973-74 
effort ; 

3) a detailed description of the procedures to be followed in all 
aspects of this year's effort; and 

4) a detailed review of the instruments to be used. 

Collection o£ Data 

Data collection took place between March and June of 1975. The 
procedures followed paralleled those of the 1973-74 effort with a site 
initially contacted by mail followed by a telephone contact made by the 
data coordinator. These initial mail and phone contacts, also described 
earlier in thi. section, were for the purpose of selection, verification 
of selection information, and scheduling of the on-site visit. Once both 
buildings for a site had been identified and scheduled for a visit, letters 

^Most of the study activities were conducted by Education Turnkey Systems, 
Inc. under contract to the Michigan Department of Education • 
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were sent to the district's director of compensatory education and the 
principals of each study school confirming these arrangements and alerting 
these persons to data needs that they might more easily fulfil) prior to 
the on-site visit. These needs were expressed as specific requests 
for enrollment data, roster of compensatory education students, and 
budget documents. 

The typical site visit included interviews with one director, two 
principals, two compensatory education teachers, six to eight regular 
classroom teachers, three or four paraprofessionals , plus one other staf£ 
for a total of 16-17 such interviews. Last year's visit, confined to 
one day, included a total of 8-9 such interviews. The time required per 
interview this year matched closely last year's experience; generally 
about one hour was required per interview. 

Onc*i a data collector completed the interviews at a given site 
and had contacted the principals and director once again to make them 
aware of this fact, all responses were coded into the keypunch columns 
of the response sheets. These completed and coded response sheets were 
then forwarded to the data coordinator for processing. At this point 
an additional request for data was mailed to each district director • 
This request for specific salary data for the persons interviewed 
on-site. Salary amounts from specific fund sources and in total were 
' requested. Also, 1973-74 test score results for the compensatory 
education students of the subject schools were requested at this time 
as well. 

Because the results of the 1973-74 study were made public prior 
to the 1974-75 on-site visits, it was felt that some measure sliould 
be taken to determine the impact of this release on the level of 
awareness of the study on the part of all persons interviewed this year. 

-10- 
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Tt would be important to know of this level of awareness in order 
to ascertain whether the results observed this year were contaminated 
by this public release of preliminary study information. The measure 
chosen was a simple question asked of each of the 808 respondents 
included in this year's study: "Are you familiar with the results 
of the first year of this study which were recently released?" The 
percent of respondents answering yes to this question is shown below 
for each type of respondent: 

1) district directors of compensatory education (48 respondents) 
— L5% said yes; 

2) principals (96 respondents) — 6% said ye^; 

3) compensatory education teachers (87 respondents) — 5% 
said yes; 

4) regular classroom teachers (356 respondents) — 2% said yes; 

5) paraprof essionals (184 respondents) — 2% said yes; and 
« 

6) other staf^ (37 respondtr : • — 5% said yes. 

tt was concluded from the above results that, below the level of district 
director of compensatory education, the study was not widely known, even 
after the public release of results and the publicity surrounding these 
results. Even at the directors* level it was not felt that the percent 
indicating awareness was high enough to cause concern over the issue of 
potential contamination. 

Searching for Missing Data Items 

All response sets were individually screened in order to identify 
all missing data items and any data inconsistencies. Once all response 
sets from a given data collector had been so screened > the data 
coordinator contacted the collector by phone and provided them their 
list of missing or inconsistent items for resolution. The data. collectors 
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then set about to resolve these problems, either correcting mistakes in 
coding or obtaining additional data from the site in question, and 
telephone their responses to the data coordinator when this was done. 
In some instances, the missing items could not be provided by tlie original 
data collector. The data coordinator resolved all such outstanding data 
problems by directly contacting the site in question. 

Reducing the Raw Data to Analyzable Form 

As described earlier in this Section, the experience of the study 
team last year had led to a revision in the data collection/coding/ 
reduction process which allowed keypunching of raw data responses to 
be done directly from the sheets filled out during the interviews. 
However, before cards could be keypunched from these forms, a number 
of steps St ill needed to be followed. Missing or inconsistent items 
liad to be resolved as was just described. Additionally, the large number 
of open-ended responses included in this year's instruments had to be 
coded for purposes of analysis. 

One of the key elements of the instrument revision process this 
year was the expansion of items found to be significant in last year's 
results. One method relied upon in this expansion or probing effort 
was the use of open-ended questions. For instance, since teacher 
morale was found to be significantly related to reading achievement, 
this year a question was added to the scaled morale response asking 
why the respondent thought morale was high (or low) in that school. 
More than 130 such open-ended items were included in the full set 
of six instruments used this year. Thus, a major task in reducing 
raw data to analyzable form was the development of codes for these 
open-ended responses and the assignment of these codes to the 
open-ended responses in each of 808 respondent sets. The codes were 
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developed l)y sampling a number of responses and obr-^ining a series 
ot" <-ode.s that fit this sample. The codes were then applied to the 
entire set of responses. For the most critical codes, a joint effort 
of the project director and the data coordinator resulted in a set of 
codes which were then field tested by two raters (or data reducers) on 
a sample of actual responses. An agreement level of 80% was set as an 
internal requirement for this field test in order to judge the 
acceptability of the codes. Where this level was not reached for a 
particular set of codes, the codes were rewritten in a manner that 
would serve to most enhance inter-rater agreements based on observed 
patterns of confusion or disagreement in the field test. Once the 
codes for all open-ended questions were developed, a staff of data 
reducers assigned a code to each open-ended response contained in the 
entire set of 808 respondent sets. 
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SECTION IV 
DESCRIPTION OF 1974-75 SAMPLE 

In this section of the report background data for the districts 
and school building included in this year's study are presented. 
The information shown in this section does not include data from 
four schools in two districts of the entire sample of 96 schools 
in 48 districts. These two sample sites (both low achieving Title I 
sites) had no program cost models built for them due to incomplete 
data. Since the discussion to follow in later sections refers to 
program cost differences across sites, it was felt that this discussion 
of background characteristics be restricted to the same sites included 
in the cost analyses* 

Comparison of Districtwide Background Data 

Table 2 shows the mean and standard deviation for each of the 
13 districtwide characteristics for the 25 successful sites and the 
2i unsuccessful sites for which program cost models could be constructed. 
Each of the 15 sets of contrasts of these background data from successful 
sites versus similar data from unsuccessful sites was tested to determine 
whether the observed differences in means between these two groups of 
sites was significant at the 0.05 level or lower. None of the comparisons 
resulted in a significant difference between the two groups of sites. 

Comparisons of School-Level Background Data 

Table 3 shows the mean and the standard deviation for each of 7 
school-level characteristics for the 50 school buildings in the study 
sample from the 25 successful sites and the 42 school buildings from 
the 21 unsuccessful sites. Each of the 7 sets of contrasts of these 
school level background data for schools from successful sites versus 
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TABLE 2 

1974-75 DISTRICTWIDE BACKGROUND DATA 
SUCCESSFUL SITES VS. UNSUCCESSFUL SITES 



BACKGROUND DATA ITEM 



UNSUCCESSFUL SITES UNSUCCESSFUL SITES 

n=25 n=21 
MEAN S.D. MEAN S.D. 



Median Family Income in 

District, Dollars Annually 

Total General Fund Expenditure 
($) per Student 

Total Compensatory Education 

Expenditure ($) per Compensatory 
Education Student (Title I for 
Title I sites, Chapter 3 for 
Chapter 3 sites) 

Number of Title I Students 

Number of Chapter 3 Students 

District Enrollment 
Kindergarten 
Grades 1-6 
Grades 7-12 
K-12 

Number of Elementary Schools 
in the District 



10,019 1622.1 



1,284 



320 
356 
310 



334 
1,979 
2,019 
4,332 



286.1 



127.8 
477.5 
541.0 



221.4 
1382.6 
1293.1 
2865.7 



9,634 1360.8 



1,153 227.8 



378 
459 
458 



313 
1,855 
1,853 
4,022 



352.0 
646.3 
652.2 



315.5 
1962.0 
1948.5 
4216.7 



A. 8 



5.9 



Number of TitJ.e I Elementary 
Schools 

Number oE Cliapter 3 Elementary 
L.i:hools 

Number of Elementary Schools 
Which Are Both Title I and 
Chapter 3 



2.9 
5.6 

3.4 



2.2 



6.7 



3.1 
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schools from unsuccessful sites was tested for significance. Only 
one of these contrasts shows a significant difference between the 
successful site schools and the unsuccessful site schools — the 
number of full time equivalent (FTE) compensatory education para- 
professionals. The schools from low achieving sites average 4.01 
FTE paraprof essionals while the high achieving sites average 2.28 FTE 
paraprofessionals. This finding is consistent with the results of 
the first year of this study. 

1974-75 Achievement Results 

As indicated earlier, a total of 96 schools in 48 sites were 

included in this year's study effort. Achievement results from the 

1973-74 school year for the sites established whether a particular 

site was included as either a low achieving or high achieving site 

for this year's effort- As part of the overall data collection effort, 

building specific 1974-75 reading achievement results were obtained. 

The data requested for each of two schools (specifically named in the 

\ 

request) per s'le were: 

1) the average gain (in grade equivalent units) for the compensatory 
education students served by the program of interest (Title I 

or Chapter 3) in reading achievement as measured by the standard- 
ized test used at that site; and 

2) the administration dates of the pre and post tests which 
determined the above average gain scores • 

These two items of data were combined into the success measure used in 

this study as follows: 

[average gain score 

month/month gain - months gained/months in the program = (converted to months) 

[number of months bet 
pre and post test (to 

^, the nearest half mont 

with a maximum value 
10.0)] 

21 
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TABLE 3 



1974-75 BACKGROUND DATA FOR SAMPLE SCHOOLS 
SUCCESSFUL SITES VS. UNSUCCESSFUL SITES 



BACKGROUND DATA ITEM 


SAMPLE SCHOOLS FROM 
SUCCESSFUL SITES 
n=50 

MEAN S.D. 


SAMPLE SCHOOLS FROM 
UNSUCCESSFUL SITES 
n=42 

MEAN S.D. 


Total Enrollment , K-6 


425 


183.0 


443 


259.0 


Total Number of Compensatory 
Education Students, K-6 


83 


72.6 


100 


79.9 


Percent of Total K-6 Enrollment 
Designated Compensatory Education 


21 


16.1 


25 


18.9 


Total Number of Regular Classroom 
Teachers, K-^b 


16 


6.2 


16 


8.8 


Ratio of Total K-6 Enrollment to 
Total Number of Regular Class- 
room Teachers, K-6 


26.9 


3.2 


28.5 


4.5 


Number of Full Time Equivalent 
Compensatory Education Teachers, 
K-6 


1.18 


0.88 


0.82 


0.93 


Number of Full Time Equivalent 
Compensatory Education Para- 
professionals, K-6* 


2.28 


2.62 


4.01 


3.20 


^ Probability of observing this large 


a difference 


by chance 


is 0.006 





'^lile data in terms of month/month gain were available for all 
A8 study sites this year on a district-wide basis and all 96 schools 
included in this year's study initially indicated that the necessary 
data for computing the above rates would be available for the 1974-75 
sciiooi year, only 41 sites (82 buildings) were able to provide the 
necessary dnta for this task. The other seven sites either did not 
fulfill the data request, had switched to objective or criterion referenced 
tests, or reported fewer test scores than was the minimum study criterion 
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for tills measure. For the 41 sites (82 schools) which reported the 
requested data. Table A shows the number of buildings reporting, mean 
and standard deviation for the month/month gain in Building 1 (nominally 
the highest acliieving building at each site which met the study's 
selection rrlterla) , Building 2 (nominally the lowest achieving 
building at each site) , and Buildings 1 and 2 combined for the following 
groupings of sites: 

1) high achieving Title I sites; 

2) high achieving Chapter 3 sites; 

3) low achieving Title I sites; and 

4) low achieving Chapter 3 sites. 

f 

A major difference in scope between last year's and this year's ^: 
study effort was indicated earlier in this section — the inclusion 
of two schools per site. Tables 4 and 5 showed the difference in 
results on the average between the Building I's and Building 2's for 
various groupings of sites. It is readily apparent from these data 
that the differences in achievement within sites (certainly within 
groups of sites) are much less than the differences between sites 
(or between groups of sites) . Table 4 even indicates that for the 
low achieving Title I sites the nominally lowest achieving schools 
(Building 2's) averaged somewhat higher gains than did the nominally 
Iiighest achieving schools (Building I's) in those sites. Thus, the 
Building 1 (high) /Building 2 (low) designation does reflect 1974-75 
reading achievement as it actually existed. In fact, viewing each 
of the six lines of Tables 4 and 5 as a possible test of whether 
any significant differences exist on the average between the Building 1 
and Building 2 results from any given site, it is noteworthy that none 
of these six contrasts indicate any significant difference between 
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TABLE 4 



1974-75 ACHIEVEMENT RESULTS (MONTH/MONlli GAIN) 
BY BUILDING DESIGNATION AND 
BY GROUPINGS OF SITES 



BUILDING 1 ONLY BUILDING 2 ONLY ALL STUDY BUILDINGS 
GROUPINGS OF SITES M MEAN S.D. N MEAN S.D. N MEAN S.D. 



High achieving Title I 
sites 


14 


1 


.88 


0 


.47 


14 


1.66 


0 


.71 


28 


1.77 


0 


.60 


High achieving Chapter 3 
sites 


7 


1 


.73 


0 


.35 


7 


1.44 


0 


.66 


14 


1.58 


0 


.53 


Low achieving Title I 
sites 


15 


1 


.01 


0 


.40 


15 


1.07 


0 


.29 


30 


1.04 


0 


.35 


Low achieving Chapter 3 
sites 


5 


1 


.20 


0 


.36 


5 


1.04 


0 


.38 


10 


1.12 


0 


.36 



TABLE 5 



1974-75 ACHIEVEMENT RESULTS (MONTH/MONTH GAIN) 
BY BUILDING DESIGNATION FOR 
HIGH ACHIEVING VS. LOW ACHIEVING SITES 





GROUPINGS OF SITES 


BUILDING 1 
N MEAN 


ONLY 
S.D. 


BUILDING 2 
N MEAN 


ONLY 
S.D. 


ALL 
N 


STUDY BUILDINGS 
MEAN S.D. 


High Achieving Sites 


21 1.83 


0.43 


21 1.58 


0.69 


42 


1.71 0.58 



Low Achieving Sites 20 1.06 0.39 20 1.06 0.30 40 1.06 0.35 



buildings. It appears that while there was a significant difference 
between the reading achievement of school districts, there was not 
significant difference between the reading achievement of the pairs 
of schools within school districts. 

With this result in mind, the possibility of using both buildings 
from any given site as a reflection of the overall success level of 
that site was Investigated. The upper right hand figures of Table 5 
show the result of combining the results from all Building I's and 
all Building 2's at high achieving sites and using the combined results 
to represent the achievement of the successful study sites. The lower 
right hand figures show the similar results in the low achieving sites. 
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SECTION V 
COST ANALYSES 

In this section of the report, the cost analyses of the compensatory 
education reading programs in this year's sample are described. Before 
specific cost data are presented, the cost methodology used in the 
study will be discussed. This discussion will allow those readers 
unfamiliar with program cost analysis co better understand the results 
that follow. Following the discussion, four analyses, all under the 
overall heading of cost analysis, are presented. The first analysis deals 
with comparisons of total program costs between the high achieving and the 
low achieving programs and between Title I and Chapter 3 programs. The 
second analysis deals with the similar comparisons but uses five separate 
subtotals which make up the total program cost (subtotals that reflect 
the cost of specific activities comprising the overall program ) as the 
basis for comparison. The third analysis deals again with similar 
comparisons but uses four other subtotals which also add to the total 
program cost (in this case the subtotals reflect the amount of resources 
from various funding sources that make up the total program cost) as the 
basis for comparison. The last of these analyses examines the 
relationship between total program cost (as well as each of the nine 
different cost subtotals alluded to above) and the month/month gain 
results presented in the previous section. 

Methodology 

This study determined the cost of all resources devoted to reading 
instruction for compensatory education students for each school in the 
study. The phrase •'cost of all resources devoted to reading instruction 
for compensatory education students'* has a very specific meaning which 
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the reader should clearly understand* An academic program, as viewed by 
an individual student, may be considered as a set of resources all 
brought to bear upon a specific objective. For this study the objective 
deals with learning to read. 

What are these resources? An obvious list of such resources 
might include: books, audio visual equipment, consumable supplies, etc. 
At least as important, though somewhat less obvious, would be the following 
resources: 

1) the time of teachers spent in the classroom actually providing 
the instruction; 

2) the time of paraprof essionals in this same regard; and 

3) the time of anyone else who actually hci§ student contact for 
this instruction. 

Even less obvious are the following resources which the student may 

or may not actually see but which are as surely devoted to this specific 

academic program as are the above items which involved student contact: 

1) staff time spent in planning the instructional program; 

2) staff time of these personnel spent in training for this 
program plus training materials or consultants; 

3) staff time spent in making the decisions necessary for the 
operation of the program — decisions on materials, classroom 
organization, training agendas, etc.; and 

4) the time of adiri-iinistrators in the overall adminis trat iye activities 
necessary for Che operation of the program. 

Each of the resources listed above has a cost associated with it; 
books and materials have prices; consultants have fees; and personnel 
are paid salaries which incur fringe benefit costs. The problem can be 
viewed as one of first identifying how much of a given resource (e.g., 
how many books, how much time) is devoted to the program and then determining 
the cost of this amount of resources by using the ••price** associated with 
that resource. For instance, if a principal devotes ten percent of his 
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time to an activity specifically related to the compensatory education 
reading program in that school, then ten percent of the salary and fringe 
benefit costs associated with the principal would be considered part of 
the total cost of the program. For ease of comparison between programs, 
the cost figure just arrived at could be divided by the number of students 
served in that building to obtain the cost per student for that resource. 

Table 6 shows the format that can be used for summarizing this 
costing process when applied to any given compensatory education reading 
program. The first column lists the potential resources that could 
be allocated to some degree to a compensatory education reading program. 
The next five columns list the activities (called "Functions" in 
Table 6) which comprise the overall program. The total amount of each 
resource allocated to each activity per student would be determined 
using a variety of cost data obtained from the district and school in 
question. A number of cells in Table 6 have been crossed out; these 
are cells which have no logical basis (e.g., consuming books and 
audiovisual software during administrative activities). Only the 
34 resource/function cells not crossed out in this table are needed to 
obtain an estiraate of the overall program cost per student (as well as 
function and resource subtotals). 

The data used for building the cost models was obtained from a 
variety of sources. Local budgets, both general fund and compensatory 
education, were obtained from each site. Resource totals used in many 
of the 34 cells of Table 6 were obtained directly from these budgets 
combined with district and school compensatory educati9n and total 
enrollment figures. In all 48 sites the appropriate compensatory 
education budget for 1974-75 was obtained; and in 47 of the 48 sites the 
1974-73 general fund budget was obtained. 
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TABLE 6 

PROGRAM COST ANALYSIS STRUCTURE 



DOLLARS PER COMP-ED 
STUDENT ANNUALLY 



RESOURCES 



a 
o 



o 
f z 

O J 



FUNCTIONS 



a 
I 

o 



a 
I 

o 



o 

M 
CO 
M 
O 

a 



o 

H 

W M 
I Z 

I g RESOURCE 
TOTAL 



< 



PERCENT 
OF 
TOTAL 
COST 



PERSONNEL 



District Comp-Ed Director 


X 


11 


17 


24 


30 


Principal 


X 


12 


18 


25 


31 


Comp-Ed Teacher 


1 


13 


19 


26 


X 


Regular Teacher 


2 


14 


20 


27 


X 


Paraprof essional 


3 


15 


21 


28 


X 


Other Staff 


4 


16 


22 


29 


32 


CONSUMABLES 












Comp-Ed Books and 












AV Software 


5 


X 


X 


X 


X 


Regular Books and 












AV Software 


5 


X 


X 


X 


X 


EQUIPMENT 












Comp-Ed AV Equipment 


7 


X 


X 


X 


X 


Other Comp-Ed Instructional 












Equipment 


8 


X 


X 


X 


X 


Regular AV Equipment 


9 


X 


X 


X 


X 


Other Instructional Equipment 


10 


X 


X 


X 


X . 


Comp-Ed Administration 












Equipment 


X 


X 


X 


X 


X 


MISCELLANEOUS 












Miscellaneous Corap-Ed 












Training Expenses 


X 


X 


X 


X 


X 


Miscellaneous Comp-Ed 












Administrative Expenses 


X 


X 


X 


X 
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FUNCTION TOTAL 
PERCENT OF TOTAL COST 
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Another major data source for this year's cost models was a salary 
listing obtained from each site for all personnel interviewed. These 
data included not only the total 1974-75 salary from all sources but 
also the specific contribution to that total from local sources, 
Title I funds. Chapter 3 funds, or other fund sources (such as the 
Section A3 state funded reading program). Salary data were obtained 
from 47 of the 48 sites for all personnel interviewed in the course of 
the on-site data collection effort. These data, coupled with the fringe 
rate data, were used to establish a "price" for each staff time resource 
identified in the study as being allocated to the compensatory education 
piogram at that school or site. 

These staff time allocation estimates were obtained in a number 
of ways. For the classroom reading activity, estimates were obtained 
from all teaching personnel interviewed as to the amount of time compensatory 
education setting, daily. Averages for these individual estimates were 
used for each program to determine the total hours of reading instruction 
received each year by each compensatory education student in that program. 
Further data from each of these same teaching personnel regarding their 
actual student contact time for compensatory education reading allowed an 
average staff ratio during reading instruction to be obtained, which 
together with the "price" data described above produced the classroom 
reading portion of these resource costs. 

For the time allocations not involving student time, each person 
interviewed provided estimates of the percent of their available time 
(i.e., working time not in contact with students) they devoted to the 
following activities: 

1) planning for compensatory education reading and other programs; 

2) training for such programs; 
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3) decision making related to such programs; and 

4) administrative duties related to such programs. 

Coupled with data on the actual amount of available working hours each 
year for that person, the total time devoted by that person to each of 
these activities was determined. Using this as a basis, the program 
total for this allocation was determined taking into consideration 
the total such persons serving the program being studied (e.g., if 
the average time devoted each year to planning for compensatory 
education rea<^ ing was 25 hours per regular classroom teacher and 10 
of the buildings 16 regular teachers served compensatory education 
students in their reading instruction, a total of 250 regular teacher 
hours was devoted to this activity yearly for that program). The cost 
of this total was then "priced** using the salary/fringe data described 
above and allocated to the number of compensatory education students 
served by that program in order to obtain further entries for Table 6. 

Once all of the cost calculations for all resources in all activities 
(or functions) were completed, the COST-ED Methodology^ produces a 
completed version of Table 6 for each compensatory education reading 
program modeled. A total of 92 such models were built; two sites 
did not provide sufficient data to allow cost models to be constructed — one 
due to a missing general fund budget, and one due to missing salary 
data — for the four schools included in the study from these two sites. 

Additionally, the salary data by fund source along with the budget 
documents allowed estimates to be made of the portion of each resource 
total included in the total program cost (the next to last column of 
Table b) form each of these four funding sources: local general fund, 
3 

A proprietary process developed by Education Turnkey Systems, Inc. 
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Tide I, Chapter 3, and other. These fund subtotals by resource were 
then summed over all resources to obtain an estimate of the amount from 
each of these fund sources reflected in the total program cost. 

In order to assess whether significant differences in total program 
costs exist in the 1974-75 study sample. Table 7 was produced by combining 
all high achieving groups of sites together and contrasting these costs 
to those for all low achieving sites combined. A comparison of total 
program costs for high achieving sites and low achieving sites shows 
that the probability of obtaining the observed difference by chance is 
.002. In other words, the total program cost for high achieving sites 
was significantly greater than the total program cost for low achieving 
sites. 

TABLE 7 

1974-75 TOTAL PROGRAM COSTS (DOLLARS PER STUDENT ANNUALLY) 
HIGH ACHIEVING VS. LOW ACHIEVING SITES 



ALL STUDY BUILDINGS 



GROUPINGS OF SITES 


N 


MEAN 


S.D. 


High Achieving Sites 


50 


635.1 


345.8 


Low Achieving Sites 


42 


458.9 


177.5 


Comparisons of Costs of 


Specific Activities 







Table 8 shows the cost results for each of five specific activities 
which make up the total compensatory education reading program. These 
activities were discussed in the methodology portion of this report and 
correspond to the first five column totals of Table 6 for each of the 
program cost models built. The results included in Table 8 are for the 
46 sites (92 school?^) for which cost data were available. The bottom 
line of Table 8 shows four of these five contrasts to be significant 

32 

-27- 



2 CO 









CO 












W M 




PU > 








CO M 












< 








O 13 




Q O 


00 


^ hJ 


u 


CO • 


-J 


H CO 


CQ 


CO > 


< 


o 




u o 




2 




M 




H > 




M W 




> h-r 




M X 




H U 
















u o 




M M 








M 




O 0(5 




W O 




PU U4 




CO 




m 




1 












ON 







SaillAIIDV 

DNiovsra 
aa dwoD Tiv 



DNI(IV3H 

aa (iHOD 

HOi SaillAIIDV 

aAiivHisiNimv 



ONicrvaH 
aa dwoD 

DNIXVK 

NoisiDaa 



DNiavan 
aa dKOD 



DNiovan 
aa JKOD 



SaillAIIDV 
WOOHSSVID 



a 

CO 



Q 

CO 



CO 
tx] 

H 

M 
CO 

o 

CO 

a 

2 
M 
O* 

O 

o 



m 

CO 

NO 



CM 
O 



iH 
O 



o 



vO 
O 
CO 



00 

to 



CM 



vo 

CM 



CM 



00 



CM 



CM 



cn 

CM 



O 



CM 
CM 



CM 
CM 



O 



CO 

o 
o 
o 



o 
o 



CM 

iH 
O 

O 



c m 

•H CM 

> n 
0) c 
•H w 

x: 

a CO 

< 0) 
00 CO 



60 rH 
C CM 

•H n 
> c 

•H 

pC CO 

a 0) 

5 CO 

o 



o 



(0 

CO 4-» 



CO 4J 
U-4 O 
O O U-l 
O 

>^ CO 
^ 0) 

•H oo a 
> u 

U 0) 
0) M-< 
CO ^ 

Pu O O 



«o 

CO 
O 



0) 

60 
60 C 
C -H 
•H > 
W 0) 
O -H 

> pC 

CO o 

^ < 



0) 

c 

CO 

o 



0) 

N to 
CO PS J3 



EKLC 



-28- 

33 



at the 0.05 level or lower. Thus, for specific planning, training, 
decision making, and administrative activities related to the compensatory 
education reading program, the high achieving sites devoted significantly 
more resources than did the low achieving sites. For classroom (student 
contact) reading activities, the high achieving sites devoted more 
resourses than did the low achieving sites; but the difference was not 
significant at the 0.05 level. 

Comparisons of Cost Provided From Various Funding Sources 

Table 9 shows the portion of total compensatory education 
reading program costs provided from each of four funding sources. These 
funding sources are local general fund monies. Title I funds. Chapter 3 
funds, other fund sources (mostly the state funded Section 43 reading 
program). The results included in this table are for the same 
46 sites (92 schools) whose program cost models have been discussed 
before in this section. In order to assess whether significant 
differences in costs provided from various funding sources exist in 
the 1974-75 study sample. Table 9 was produced by combining all high 
achieving groups of sites and contrasting these costs from various 
funding sources with those for all low achieving sites combined. The 
mean values shown in the first four columns of Table 9 add to the mean 
values shown in column five. Three of the four fund categories considered 
in this table show the high achieving sites have significantly 
(at the 0.05 level) larger amounts of program costs provided from 
these funds than in the case in the low achieving sites. Only the 
costs provided from Chapter 3 funds show no difference between the high 
and low achieving sites. Actually more Chapter 3 funds are allocated to 
the compensatory reading programs in low achieving sites in the study 
saoiple though the difference is not statistically significant. 
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TiiL» results siiown in this table — chat high achieving compensatory 
education programs receive significantly higher allocations of general 
funds, Title I funds, and funds from other fund sources (other than 
Chapter 3 funds) are especially interesting in light of the analysis 
previously presented. In that previous analysis it was pointed out that 
the total general fund expenditures per student annually in the high 
achieving sites, while somewhat greater than those in low achieving 
sites, were not significantly different from those in low achieving 
sites. Yet, at the program level, the analysis presented indicated that 
within this overall funding context, where no significant differences 
were found, significantly more local general funds were allocated to 
the compensatory education reading program in the high achieving sites 
than in the low achieving sites. These two results are important for 
they indicate that different allocation patterns within a general 
context of equal total funding may produce differential program results; 
i,e., more important than how much money is spent overall is the actual 
amount allocated to a specific program. The data for each of the two 
years of this study indicate that where significantly more resources 
are allocated to a specific program (in this case compensatory education 
reading) , the districts having approximately equal background levels of 
funding, significantly greater achievement in reading for compensatory 
education students are also observed. 

The results shown in Table9 regarding Chapter 3 funding reflects 
the nature of the Chapter 3 program. Given a fixed ceiling of $200 per 
student, a much lower level of funding than Title I, wide variations 
in allocations of Chapter 3 funds are less possible, thereby making 
significant differences less likely to be observed. The fact that low 
achieving sites in the study sample actually were allocated slightly 

36 

-31- 



higher levels of Chapter 3 funds also reflects the nature of the 
Chapter 3 program — concentration of funding based on educational 
need. 

The results shown in Table 9 regarding other funding sources is 
explained as follows. The most of these other funds were from the 
state funded Section 43 reading program. These funds are available 
to Title I eligible schools which do not receive Chapter 3 funds. Thus, 
the increased level of allocation of these funds in the high achieving 
sites mirrors the decreased allocation of Chapter 3 funds just described. 

Relationship Between Program Costs and Observed 1974-75 Achievement Scores 

The cost analysis presented so far in this section dealing with 
high achieving vs. low achieving groups of programs were based on 
program success designations reflecting 1973-74 achievement data for 
these programs. However, another possible way of examining the relationship 
between program cost in 197 4-7 5 and program success in 1974-75 is to 
directly explore the relationship between the two. Program cost results 
for 92 of the 96 programs studied have just been presented and analyzed 
in some detail. Considering only those programs for which both cost 
results and achievement results were available for the 1974-75 school year, 
a total of 80 programs provided such data. 

Figure 1 is a scatter diagram of the achievement results (vertical 
axis) and program results (horizontal axis) for these 80 programs. Note 
the legend showing the four groupings of programs reflected in this exhibit: 

1) high achieving Title I sites; Q 

2) high achieving Chapter 3 sites; D 

3) low achieving Title I sites; and ^ 

4) low achieving Chapter 3 sites. BBI 
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AIko sliown in Figure 1 is the least squares regression line which 
best fits tiiese 80 points. This line, for the 1974-75 study is 
shown as a solid line. The dotted line also shown in this figure 
represents the regression line which best fits the 48 data points 
available from last year's study. The equations for these two 
regression lines are as follows: 

1974-75 

month/month gain - 1.0016 + 0.000774 (total program cost) 
(r = 0.2913; regression coefficient is significant at the 
0.004 level) 

1973-74 

month/month gain - 0.697 + 0.000941 (total program cost) 

(r = 0.5514; regression is significant at the 0.00005 level) 

Thus, it can be seen that while the slope of this year's regression 
line is not as steep as last year's, a significant relationship between 
month/month gain and total program cost exists in this year's data 
paralleling a major finding of last year's study effort. Further, 
there is not significant difference between the slopes of the two 
regression lines shown in this figure, though this year's line is 
somewhat less steep. 

As a further analysis, the relationships between each of the 9 
major subtotals of total program cost discussed earlier in this 
section and month/month gain were also investigated. Table 10 shows 
the Pearson correlation coefficient between each of these cost subtotals 
and compensatory education reading achievement for the 1974-75 data. 
The correlation result for total program costs have been included in 
this figure as well for reference purposes. Three of the five cost 
subtotals dealing with specific activities show a significant positive 
correlation with program achievement results: classroom reading activities, 
planning, and decision making. Note that decision making shows the 




highest correlation of these three activities but chat none of the 
correlation coefficients for these specific activity cost subtotals 
is larger than the correlation for total program cost. 

TABLE 10 



CORRELATIONS BETWEEN MAJOR COST SUBTOTALS AND 
MONTH/MONTH GAIN ACHIEVEMENT RESULTS 
FOR THE 1974-75 STUDY SAMPLE 





r (between 






month /month 






gain and 




COST SUBTOTAL 


cost sub- 






total) 


2 

r 


Total Program Cost 


+0.2913* 


0.0848 


Specific Activity Cost: 






Classroom Reading Activities 


40. 2019* 


0.0407 


Planning for Comp. Ed. Reading 


+0.2091* 


0.0437 


Training for Comp. Ed. Reading 


+0.0795 


0.0063 


Decision Making for Comp. Ed. Reading 


+0.2753** 


0.0757 


Administrative Activities for Comp. Ed. Reading 


+0.1094 


0.0119 


Costs Provided from Various Funding Sources: 






Local General Fund 


+0.0822 


0.0067 


Title I Funds 


+0.3629** 


0.1316 


Chapter 3 Funds 


-0.1000 


0.0100 


Other Fund Sources 


+0.0703 


0.0049 



*Significant at the 0.05 level but not at the 0.01 level. 
**Signif icant at the 0.01 level or lower. 



Only one of the four cost subtotals dealing with costs provided from 
various funding sources shows a significant positive correlation with 
program achievement results — Title I funding allocation.. Note that 
this particular correlation coefficient is larger than that for total 
program cost. 
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SECTION VI 
EFFECTIVENESS ANALYSES 

Tn this section of the report, the results of the effectiveness 
analyses conducted using the 1974-75 achievement data are described. 
The purpose of these analyses was to identify those conditions, 
activities, or things (policy variables) which are related to student 
reading; achievement. While the previrn*: section indicated that the 
atnount of resources allocated to reading instruction was related to 
student reading achievement, it was actually those conditions, 
activities, and things purchased by the resources that were related to 
the student reading achievement. 

The effectiveness analyses can be separated into two general 
groups. The first group is used to determine which of the policy 
variables under investigation discriminated between the high and low 
achieving sites. The product of this first group of analyses is a set 
of variable's which significantly dllscri^minated between high and low 
sites. The second group of analyses is concerned with each of the individual 
variables that were identified by the first group of analyses. The 
various sites are divided into either: 1) two groups reflecting a high 
degree or a low degree of th'e policy variable being studied (in the case 
of quantitative variables) or 2) groups representing the various categories 
of the policy variable (in the case of qualitative variables). The reading 
achievement gains observed for these various groups were then examined to 
determine if reading achievement varied according to these groups. In this 
way, the data gathered during any given year of the Cost-Effectiveness Study 
was examined twice. 
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Ourin;j, thct first year (1973-74) of the study, 45 policy variables 
were found to discriminate between high and low achieving sites* The 
hi^h and low sites for this first year of the study were identified using 

1972- 73 reading achievement data. The second step of the first year 
analyses was to determine if the 45 identified variables were significantly 
related to the 1973-74 reading achievement gains at the .05 level. It 

was found that 17 of the 45 variables were significantly related to the 

1973- 74 reading achievement gains. In other words, the first year of ;:he 
study identified 17 policy variables that were cross-validated over two 
years of reading achievement results. 

The effectiveness analyses for the second year of the study (1974-75) 
were conducted in the same manner as in the first year of the study. There 

one modification in the analysis plan for the second year. The pre- 
f^stablished significance level for Che first year of the study was .05 
for both sets of analyses. For.;tthe second year, the significance level 
for the first set of analyses, a general screening procedure, was initially 
set at .10 rather than .05.^ The significance level for the second set of 
analyses remained at .05. 

Cross-validation of First Year Resul ts 

The first step of the second year analyses was to examine those 
variables which were found to be significant during the first year of the 
study. These variables included 37 of the 45 that were found to be 
significant during the first phase of the first year analyses^ Also 
included in this analysis were those variables which were significantly 
relitcd to achievement on the first phase of the first year analyses 

^At this point* the significance level was changed back to .05 
because the number of variables significant at the ,10 level becnmc 
unmanageable. 
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but not on the second phase of the first year analyses. It was found 
Chat 5 of the 37 variables included in the analysis discriminated 
significantly between the high and low achieving sites (site selection 
based on 1973-74 achievement results). These variables are listed 
in Tables 11. . 

The next step in the analysis^was to examine the 683 new/modified 
variables (i.e., those variables not included in the first year of the 
Study) to determine if they discriminate between the high and low sites. 
This analysis showed there were 120 variables which discriminated 
between high and -low sites at the .05 level or better. These variables 
are listed in Appendix A. 

TABLE 11 

EDUCATIONALLY SIGNIFICANT VARIABLES 



Variable 



Relationship to Reading 
Achievement 



Principal (respondent) 

1. Number of teacher 
working hours at 
school daily 

Compensatory Education 
Teacher (respondent) 

2. Fraction of materials 
selected by teacher 



3. Days of training 

provided teachers at 
onset of project 

A. Did paraprof essionals 



High number of hours was associated 
with high reading achievement 



High fraction of materials selected 
was associated with high 
reading achievement 

Greater number of days was associated 
with high reading achievement 



5. Teacher morale 



Except where paraprof essionals 

functioned as second teachers, 
assistance from paraprofessionals 
was associated with low reading 
achievement 

High teacher morale was associated 
4 3 with high reading achievement 
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The next step in the analysis was a phase two analysis for the 
second yc^ir; this analysis attempted to find out if groups based upon 
the variables that were significantly related to achievement in previous 
analyses were different in terms of 1974-75 reading achievement scores. 
Of these 120 variables, it was found that 36 were significantly related 
to the 1974-75 reading achievement scores. These variables were then 
examined (using non-statistical methods) to determine if there were 
any interrelationships within them. It was found that there were five 
groups of variables that were dealing with this same concept or topic. 
Those groups are listed in Table 12. 



TABLE 12 

CATEGORIES OF VARIABLES 
RELATED TO READING ACHIEVEMENT 



Category 


Relationship to Reading 
Achievement 


Degree to which 


High degree of implementation was 


accountability was 


associated with high reading 


implemented 


achievement 


Extent of parapro- 


High level of involvement was 


f ess ional involvement 


associated with low reading 




achievement 


Involvement of private 


Private firm involvement was 


I irms other than selling 


associated with low reading 


materials 


achievement 


Degree of program 


High degree of program organization 


organization 


was associated with high 




reading achievement 


Professional staff 


(A single relationship charac- 


involvement 


terizing this group of varinbJes 




has not been identified) 
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For those variables that were included in the first year of the 
study, it was possible that they would be included in four separate 
analyses: phnse one and two for both year one and two. The question 
arose, how many analyses would be necessary for a variable to be 
significantly related to reading- achievement before 'it could be said 
that it was an important variable with respect to reading achievement? 
Certainly, a variable which was significantly related to achi^^-vemunt 
of all four variables could be considered important. But would a 
variable which showed significant relationship only two of the four 
times he consici-red important? What about a variable which was related 
three times? These questions can be summarized into one question, what is 
the power of the statistic used to identify their relationship? Most 
statistical tests are performed in such a way to give a probnbility 
statement as to the chances of finding significance when no real 
significance exists. The power of any tests deals with the probability 
of not finding significant relationship when such a relationship does 
exist. Since the t-test was used more than any other given statistic 
and means of calculating its power are readily available, it was 
decided to examine the power of the t-test given the population scores 
with which the study deals. It was found that the power of most of the 
t-tests, for the .05 l«vel, range from approximately .6 to .7 with few 
examples going above .75. These figures had an average of approximately .67. 

The implication of this result is that it could be expected that if 
a true relationship existed between some variable and achievement, the 
typical t-test used in the study would detect that relationship only two 
out of three times. It was decided at that point to include as ed'i.-ationally 
significant variables any variable which was found to be related to achieve- 
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ment Ihri't- out oi four analyses. This is a conservative choice in that 
three out of four is above the two out of three ratio estal^lishod above. 



TABLE 13 



VARIABLES RELATED TO READING 
ACHIEVEMENT ON THREE OF FOUR ANALYSES 
DURING 1973-74 AND 1974-75 STUDY YEARS 



Variable 



Relationship to Reading 
Achievement 



Principal (respondent) 
1 • Number of teacher 
working hours at 
school daily 

2. Principal's satisfaction 
with regular teacher's 
coordination, of 
reading services, with 
compensatory education 
reading services 

Compensatory Education 
Teacher (respondent) 

3. Fraction of materials 
selected by teacher 



4. Days oi training 
provided teachers at 
onset of project 

5. Did paraprof essionals 
help teacher 



6. Teacher morale* 



High number of hours was associated 
with high reading achievement 



High level of satisfaction was 
associated with high reading 
achievement 



High fraction of materials selected 
was associated with high 
reading achievement 

Greater number of days was associated 
with high reading achievement 



Except where paraprof essionals 

functioned as second teachers, 
assistance from paraprof essionals 
was associated with low reading 
achievement ' 

High teacher morale was associated 
with high reading achievement 



Those variables included in both years of the study were examined so 
as to determine how many times they were shown to be significantly related 
to reading achievement* Those variables which were significantly related 
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to achievement at least on three of the four analyses are included in Table 13. 

It £ihouId be noted at this point that only one variable was related to 

achievement on all four analyses; that variable is noted with an asterisk 
in the TnbLe. 

T:ibles 12 and 13 contain 11 variables or groups of variables which 
have been found over one or both years of the study to be significantly 
related to rending achievement. The remainder of the effectiveness analysis 
is concerned only with these 11 variables or groups of variables. The 
number of variables considered was actually reduced to ?• The group of 
variables representing professional staff involvement was dropped because 
no single measure characterizing this group of variables has been identified; 
continuing analyses will address this problem. A group of variables concerning 
the extent of paraprof essional involvement was dropped in favor of the 
variable in Table 11 representing the nature of paraprof essional involvement. 
Finally, the variables, in Table 11, pertaining to the number of hours 
teachers spend at school daily and the fraction of materials selected by 
the compensatory education teacher were dropped. These variables were 
dropped because they were not significantly related to reading achievement 
on the fourth and last of the analyses. The data, primarily correlation 
coefficients, for the remainder of the effectiveness analyses was obtained 
from the fourth series of analyses. Without a significant correlation 
between these variables and reading achievement it was decided not to use 
them in the analysis. 

Thus, the final set of analyses was perfonned on 8 variables including 
reading achievement gains. These variables and their correlations are shown 
in Table 14. 



TABLE 14 



iNTERCORRELATlON MATRIX FOR VARIABLES 
INCLUDED PATH ANALYSES 





Variable 






Variable Number 






Variable 


Number 


2 




/. ^ A 
^ D O 


7 
/ 


Q 

o 


Days training* 


1 


-.17 


-.19 


-.05 .00 .07 


.11 


.21 


Paraprof essional 
Involvement 


2 




-.12 


-.17 .12 -.08 


-.04 


.20 


Teacher 
Morale 


3 






.18 .12 .42 


.24 


.28 


Private 

Involvement 


4 






-.31 -.20 


.03 


-.18 


Degree of 

Accountability 


5 








. UU 




Degree of Program 
Organization 


6 








.10 


.22 


Principal Satis- 7 
faction with Teacher 
Coordination 










.31 


Reading 

Achievement 


8 








% 





* See Tables 12 and 13 for more detailed descriptions of variables. 



Based on previous analyses, it is known that the variables in Table 14 
are related to reading achievement. However, the previous analyses do not 
provide an indication as to how those variables are related to reading 
achievement and to each other. To determine the nature of these relationships, 
it was decided to use path analysis as described in Multiple Regression and 
Behavorial Research , Kerlinger and Po.dhazur. It should be noted at this point 
that path analysis is not a means of identifying causes but a means of 
determining which of various proposed explanations of reality be«t fit the 
data at hand. 
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The path model shown in Figure 2 was developed from the correlations 
shown in Table 14. It was the first path model for which a complete path 
analysis was calculated. Notice that the arrows between the various boxes 
represent the hypothesized directions of the relationships between the 
variables. For example the arrow between variable 3, teacher morale, and 
variable 8, student achievement, indicates that it was hypothesized that 
teacher morale affected student achievement. The correlation between teacher 
morale and student achievement was .28. This is shown alongside the 
arrow connecting those two variables. The figure in the parentheses below 
the correlation of .28 is the reproduced correlation. The degree to which 
the hypothesized path model reflects the reality of the data at hand is shown 
by the agreement between the correlation and the reproduced correlation in 
the parentheses. For certain relationships between the variables, the path 
coefficient was not calculated. For example, no path coefficient was 
calculated for the relationship between variable 4, presence of private 
firms, and variable 5, degree of implementation of accountability. Where 
only one variable is hypothesized to affect another variable (only variable 4 
was hypothesized to affect variable 5) , the correlation between the two 
variables is identical to the path coefficient between the two variables. 

Model 5A, shown in Figure 2, shows that the agreement between the 
correlation and reproduced correlations between the variables of teacher 
morale, preservice training, and paraprof essional involvement and the 
variable achievement were not high. This disagreement between the correlations 
and the reproduced correlation indicates that Model 5A is not a good 
representation of reality as represented by the data at hand. 
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PRIVATE 
FIRM 
INVOLVEMENT 



DEGREE OF 
ACCOUNTABILITY 




DEGREE OF 
PROGRAM 
ORGANIZATION 



.42 



TEACHER 
MORALE 



READING 
ACHIEVEMENT 




DAYS OF 
PRESERVICE 
TRAINING 



(-.17) 



INVOLVEMENT OF 
PARAPROFESS lONALS 



..31 



PRINCIPAL'S 
SATISFACTION 

WITH COORDINATION 
BETWEEN REGULAR 
AND COMPENSATORY 

EDUCATION TEACHERS 



Figure 2 
Path Analysis Model 5A 
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PRIVATE 
FIRM 
INVOLVEMENT 



DEGREE OF 
ACCOUNTABILITY 




DEGREE OF 
PROGRAM 
ORGANIZATION 



,42 
(.41) 



TFJVCHER 
MORALE 



.28 
.(.27) 



READING 
ACHIEVEMENT 



.19 



(-.19) 



DAYS OF 
PRESERVICE 
TRAINING 



.20 



(.21) 



-.17 



INVOLVEMENT OF 
PARAPROFESSIONALS 



.31 



PRINCIPAL'S 
SATISFACTION 
WITH COORDINATION 

BETWEEN REGULAR 
AND COMPENSATORY 
EDUCATION TEACHERS 



Figure 3 
Path Analysis Model 12 
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PRIVATE 
FIRM 
INVOLVEMENT 



DEGREE OF 
ACCOUNTABILITY 




DEGREE OF 
PROGRAM 
ORGANIZATION 



.A2 
(.A2) 



TEACHER 
MORALE 




DAYS OF 
PRESERVICE 
• TRAINING 



-,17 



READING 
ACHIEVEMENT 



y 



,20 



(.22) 



INVOLVEMENT OF 
PARAPROFESSIONALS 



i 



.31 



PRINCIPAL'S 
SATISFACTION 
WITH COORDINATION 
BETWEEN REGULAR 
AND COMPENSATORY 
EDUCATION TEACHERS 



Figure A 
Path Analysis Model 14 
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While many models were examined, the next model presented in Figure 3 
showed a high degree of agreement between the correlation coefficients and 
their respective reproduced correlations. While Model 12 did represent the 
best fit with the data at the point in time it was developed, further path 
analyses were performed. Model 14 in Figure 4 represents a minor modification 
of Model 12. As can be seen from the degree of agreement between the 
correlations and reproduced correlations in Model 14 and those in Model 12, 
Model 12 does represent a better fit with the data at hand than does 
Model 14. Because Model 12 is less complex than Model 14 and is a better 
fit with the data at hand than Model 14, it was selected as being a 
superior model. 

At this point, it was decided that Model 12, presented in Figure 3, 
represented a sufficiently accurate model of reality as represented by the 
data available.^ This is not to say that further analyses would not find 
a model which would better fit the data. Moreover, it is now a challenge 
to demonstrate, through systematic analysis, that there is another model 
which better fits reality. 



5 

Fifteen path models were analyzed. Models 5A and 14 were the 
most complex models considered^ g 

-48- 



SECTION VIE 
SUMMARY, CONCLUSIONS, AND RECOMMENDATION 

Summary 

The purpose of this report is to provide a ciesi:r iption of the 
1974-75 Michigan Cost-Effectiveness Study and its findings. As way 
the case with the first year of the study, 1973-74, the 1974-75 study 
was restricted to compensatory education reading programs. The study 
was an effort to develop and implement evaluation techniques which can 
determine what educational practices bring about changes in student 
behavior and what costs are associated with those practices. 

The Cost-Effectiveness Study has focused upon educational variables 
which could be changed or controlled by educational systems. Variables 
such as race, social economic status, level of parental education, and 
so forth, which cannot be readily controlled or modified by an educational 
system were not examined. It was the intent of the study to examine 
those aspects of educational policy and practices which could be changed 
to bring about a higher quality of education for students. 

The design implemented for the first year of the Cost-Effectiveness ^ 
Study consisted of two components. An effectiveness component was 
developed and integrated into the cost component derived from the 
COST-ED model. The criterion for measuring effectiveness was grade 
equivalent gains measured on a month per month in program basis using 
standardized norm-referenced tests administered by participating districts. 
Process variables were used as independent variables and, through 
various analytical techniques, were contrasted between successful and 
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unsuccessful programs to dtrtermine if any significant relationship 
existed. The COST-ED model was modified and used to determine the 
costs associated witli variables and practices significantly related to 
proj^ram success. 

The overall purpose of the 1974-75 study was the continued development 
of the analytical techniques reflected in the cost-effectiveness model. 
However, a critical part of 'this continued development was considered 
to be a cross-validation effort focusing upon the variables identified 
and reported in the executive summary of the 1973-74 study. The 
importance of cross-validation rests in the need for evidence upon 
which CO anchor the overall findings, evidence that involves the 
demonstration ol significant results over more than one year of the study. 

Following, in importance, the cross-validation of the reported 
results of last year's effort is the identification of new variables 
which relate to achievement. 

The third and last purpose of the 1974-75 effort was the investigation 
of the direction of the relationships between achievement and various 
identified variables. The development of the path models presented in 
Section VI do represent a major step in Identifying the nature of the 
significant relationships between various variables and reading achievement. 

For tlie 1974-75 study year, a number of changes in the overall 
study design were carried out. Two buildings per study site were included 
in tlje sample, both the highest and lowest achieving buildings from 
eac li district Csite) regardless of whether the distriiut was selected 

n \ih\h or low ;i(^hievlng site. This basic change, from 1973-74, in 
Llie design was carried out in order to investigate within school district 
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variables. Btrcause of this change in design, districts having only 
one elementary school were included in the study. Thus, the study 
sample for the 1974-75 study year tended to include distri*.-ts whu'h 
on tiie average were iar);cr than the previous year's study sample?. 

Following the overall dimensions intended for the 1973-74 study, 
the site selection process resulted in identifying 25 districts in 
Michigan highly successful in their compensatory education reading 
programs and 25 that were highly unsuccessful. Thirty six of these 
50 districts were to be included for their Title I programs; fourteen 
for their Chapter 3 programs. 

Data collection took place between March and June of 1975. The 
procedures followed paralleled those of the 1973-74 effort. Once 
both buildings for a site has been identified and scheduled for a 
visit, letters were sent to the district's director of compensatory 
educacilon and the principals of each study school confirming these 
arrangemtents and alerting these persons to data needs that they might 
mure easiily fal;ill prior to the on-site visit. These needs were expressed 
as spec /Tic requests for enrollment data, roster of compensatory education 
student's, and budget documents. The typical site visit included 
interviews with one director, two principals, two compensatory education 
teachers, six to eight regular classroom teachers, three or four 
paraprofessionals , plus one other staff for a total of 16-17 such interviews. 

The following background characteristics were compared for successful 
and unsuccessful sites: 

1) Median Family Income in District, Dollars Annually 

2) Total General Fund Expenditure ($) per Student 

/J) Total Compensatory Education Expenditure ($) per Compensatory 
Education Student (Title I for Title I sites. Chapter 3 for 
Chapter 3 sites) 
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4) Number of Title I Students 

5) Number of Chapter 3 Students 

6) District Enrollment 

7) Kindergarten 

8) Grades 1-6 

9) Grades 7-12 

10) K-12 

11) Number of Elementary Schools in tUo District 

12) Number of Title I Elementary Schools 

13) Number of Chapter 3 Elementary Schools 

14) Number of Elementary Schools Which are Both Title I and Chapter 3 
There were no significant differences between the two groups of sites on 
these c-hararteristics . 

A major difference in scope between the 1973-74 and 1974-75 study 
was the inclusion of two schools per site: one high achieving school and 
one low achieving school. It was found that the difference in acheiveraent 
between sites was significant but that the comparison of all high achieving 
schools (regardless of site designation) to all low achieving schools 
showed no significant difference in reading achievement. With this 
result in mind, buildings from high achieving sites were designated 
high achieving, buildings and buildings from low achieving sites were 
designated low achievinp, buildings. 

Four cost analyses were presented. The first analysis dealt with 
comparisons of total program costs between the high achieving and the 
low achieving programs. The second analysis dealt with similar comparisons 
but used five separate subtotals which make up the total program cost 
(subtotals that reflect the cost of specific activiti es comprising the 
overall program ) as the basis for comparison. The third analysis dealt 
again with similar comparisons but used four other subtotals which also 
add to the total program cost (in this case the subtotals reflect the 
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amount of resources from various funding sources that make up the total 
program cost) as the basis for comparison. The last of the cost 
analyses examined the relationship between total program cost (as well 
as each of the nine different cost subtotals described above) and the 
month/month gain results presented in the previous section. 

This study determined the cost of all resources devoted to reading 
instruction for compensatory education students for each school in the 
study. The phrase "cost of all resources devoted to reading instruction 
for compensatory education students" has a very specific meaning which 
the reader should clearly understand. That meaning is that any cost 
incurred, by school district staff, performing the functions of compensatory 
education reading: 1) instruction, 2) planning, 3) training, 4) decision 
making, and 5) administration. 

A comparison of total program costs for high achieving sites and lev/ 
achieving sites showed that the total program cost for high achieving 
sites was significantly greater than the total program cost for low 
achieving sites. The cost per student for high achieving sites was $635 
as compared to $459 for Lh,e low achieving sites. The cost analyses showed 
that for the function of planning, training, decision making, and 
administrative activities related to the compensatory education reading 
proj^ram, the high achieving sites devoted significantly more resources 
than did the low achieving sites. For classroom" (student contact) 
reading activities, the high achieving sites devoted more resources 
than did the low achieving sites; but the difference was not statisticall 
significant • 
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'riiree of the four fund sources examined showed high achieving sites 
allocated significantly larger dollar amounts per pupil than did low 
achieving sites. Only the costs provided from Chapter 3 funds show no 
difference between the high and low achieving sites. Actually more 
Chapter 3 funds were allocated to the compensatory reading programs in 
low achieving sites in the study sample though the difference is not 
statistaically significant. 

The cost analyses presented in the first three sets of analyses 
dealt with high achieving vs. low achieving groups of sites, based on 
program success designations reflecting 1973-74 achievement data for 
those sites. Comparing program costs to 1974-75 reading achievement 
resulted in finding that the following cost categories (in terms of 
annual per pupil cost) were significantly related to reading achievement: 

1) Total Program Cost 

2) Cost of Classroom Reading Activities 

3) Cost of Planning for Compensatory Education Reading 

4) Cost of Decision Making for Compensatory Education Reading 

5) Title I Funds 

In all five cases, higher costs were associated with high reading achievement. 

The purpose of these effectiveness analyses was to identify those 
conditions, activities, or things (policy variables) which are related 
to student reading achievement. While the cost analyses indicated that 
the amount of resources allocated to reading instruction was related 
to student reading achievement, it was actually those conditions, 
activities, and things purchased by the resources that were related 
to the student reading achievement. 
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The effectiveness analyses was separated into two general groups. 
The first group was used to determine which of the policy variables 
under investigation discriminated between the high and low achieving 
sites. The second group of analyses was concerned with each of the 
individual variables that were identifier by the first group of analyses. 
The various sites were divided into either: 1) two groups reflecting 
a high degree or a low degree of the policy variables being studied 
(in the case of quantitative variables) or 2) groups representing 
the various categories of the policy variable (in the case of 
qualitative variables). The reading achievement gains observed 
for these various groups were then examined to determine if reading 
achievement varied according to these groups. In this way, the data 
gathered during any given year of the Cost-Effectiveness Study was 
analyzed twice. 

The results ot the first yea:: of the Cost~Ef f ectivenses Study 
showed that 45 variables varied significantly between the high end 
low study sites, -ince these sites were selected on the basis of 
1972-7:3 reading achievement data, it was desired to determine if any 
of these 45 variables were related t.^ the 1973-74 reading achievement 
r»-sults. Further analyses iudicated that 17 of the 45 variables were 
significantly related to the 1973-74 reading, achievement results. 
Tne 1974-75 data analyses examined, wtiere possible, the 45 variables 
from the first year cf the study to determine if they were related 
to reading achievement based upon the second year results. These 
variables --ore examined with respect to the high and low achieving 
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sites (which were based upon 1973-74 achievement data) and with respect 
to the 1974-75 reading achievement. Thus, over a period of two years, 
there were four separate analyses which examined the relationship between 
the 45 variables previously mentioned and reading achievement. 

Those variables which were found to be significantly related 
to reading achievement in three of the four analyses were considered 
to be variables whose relationship to reading achievement were considered 
to be educationally significant. 

In addition to the 45 variables discussed above, there are variables 
which were modifications of variables used in the first year of the study 
and variables that were completely new to the study in the second year. 
These variables were analyzed by determining if they were related to high 
and low /achieving sites and then also to determine if they were related 
to 1974-75 achievement results. Thus, these variables were analyzed 
only twice. Of these variables, it was determined to focus upon those 
which were found to be significantly related to reading achievement 
for both of the analyses. There were 36 such variables. An examination 
of the 36 variables indicated that there were five general categories 
into which most of these variables could be classified. 

Eleven variables or groups of variables were found, over 
one or both years of the study, to be significantly related to reading 
achievement. The remainder of the effectiveness analysis was concerned 
only with these 11 variables or groups of variables. The variable 
representing professional staff involvement was dropped because no single 
measure characterizing this group of variables has been identified. A 
group of variables concerning the extent of paraprof essional involvement 
was dropped in favor of a variable representing the nature of para- 
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professional involvement. Finally, the variables pertaining to the number 
of hours teachers spend at school daily and the fraction of materials 
selected by the compensatory education teacher were dropped. These 
variables were dropped because they were not significantly related to 
reading achievement on the fourth and last of the analyses. The data, 
primarily correlation and coefficients, for the remainder of the effectiveness 
analyses was obtained from the fourth series of analyses. Without a 
significant correlation between these variables and reading achievement 
it was decided not to use them in the path analysis. Thus, the final set 
of analyses was performed on eight variables including reading achievement 
gains . 

To determine the nature of these relationships between those 
eight variables, it was decided to use path analysis as described 
in Multiple Regression and Behavioral Research, Kerlinger and Pedhazur. 
Path analysis is not a means of identifying causes but a means of 
determining which of various proposed explanations of reality best fit 
the data at hand. Path analysis could yield different results with a 
different set of data. Path analysis does assist the researcher in 
determining which of various explanations best fit the available data. 

While many models were examined, one model was shown to possess 
a high degree of agreement between the correlation coefficients and 
their respective reproduced correlations. While Model 12, presented in 
Figure 3, did represent the best fit with the data at the point in 
time it was developed, further path analyses were performed. However, 
no other model was found which gave a higher degree of agreement between 
the correlation coefficients and their respective reproduced correlations. 
It was decided that Model 12 represented a sufficiently accurate model 
of reality as represented by the data available. This is not to say 



Lnac rurcner analyses wouxa noc rmd a modex wnicn wouxd Detter tit 
the data. Moreover, it is now a challenge to demonstrate, through 
systematic analysis, that there is another model which better fits 
reality. 



Conclusions 

1. A set of procedures have been developed which when implemented 
can: 

a) identify those costs, in terms of resources used, which 
may be associated with a desired educational product; 

b) identify those policy variables which may be associated 
with a desired educational product ; and 

c) identify the most plausible model(s) which describes 
the nature of the association between the policy 
variable and educational product. 

2. It has been shown that for extremely high and extremely low 
achieving compensatory education reading programs in Michigan: 

a) higher per pupil program costs are associated with a higher 
reading achievement; and 

b) certain policy variables, see Figure 4, page 47, do appear 

to have significant impacts on student compensatory education 
reading achievement. The variables Teacher Morale, Degree 
of Accountability, and Involvement of Paraprof essionals 
were the only variables which were shown to have direct 
impact on student reading achievement. 

Re commend at ion 

It is recommended that the results of the Michigan Cost-Effectiveness 
Study be implemented on a pilot basis in a small number of schools where 
student reading achievement is low. The goal of the pilot implementation 
would be to determine if student reading achievement can be raised as a 
result of changes based upon the Cost-Effectiveness Study results. 
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